Shoestring Evaluation: Designing Impact Evaluations Under Budget, Time, and Data Constraints

This paper discusses the shoestring evaluation approach, which was developed to assist evaluators in conducting evaluations that are as methodologically sound as possible when operating with budget and time constraints and with limitations on the types of data to which they have access. The approach is used in two main scenarios. In the first scenario, the evaluator is not called in until the project or program has been operating for some time and typically no baseline data had been collected on the project population or a control group. As managers, policy makers, and funding agencies often only start to focus on assessing impacts when the time to make decisions on future funding is approaching, the evalua-tor is frequently required to work without an adequate timeline and often with a limited budget. In the second scenario, the evaluator is called in at the start of the project but for budget, political, logistical, or methodological reasons it was not possible to collect baseline data on a control group, or in some cases even on the project population itself, using methodolo-gies that will be comparable to later evaluation. In response to the growing demand for evaluation with these budget and time constraints, a number of rapid and economical assessment methods have been developed. Unfortunately , in an effort to deliver evaluation results on time and within budget, many of the basic principles of sound evaluation design, such as random sampling, specification of the program theory, instrument development, control for researcher bias, and general quality control may be compromised. The shoe-string evaluation approach provides tools for working within limitations of budget, time, and data, while at the same time providing a framework for the identification of threats to the validity or adequacy of the evaluation findings, and guidelines for addressing the different threats once they have been identified. The approach was originally developed to assist evalua-tors working in developing countries, where the budget, time, and data constraints are often most severe; but feedback from colleagues working in the U.S. and other industrial nations suggests that the approach may be more widely applicable. However, all of the case studies in the paper are taken from the authors' experience in developing countries. Most of the tools and methods used in the approach will be familiar to experienced evaluators. What is new is the way the tools are combined into a six-step strategy to ensure the best …